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ABSTRACT 

Theoretical and empirical woik was conducted on the 
role of the "mental model," or how-it-works information, in learning 
to operate equipment. The original project was concerned V7ith 
empirical and cognitive modeling studies of how people learn to 
operate equipment from the kind of information contained in technical 
documentation. The goal was to understand how knowledge about 
equipment could be presented effectively and how knowledge about the 
equipment should be conveyed to the reader. These studies v;ere 
described in prior publications, which are listed in this final 
report. The second part of the project was a separate line of work on 
a computerized aid for comprehensible writing, a computer program 
intended to provide feedback to the writers of technical documents 
about comprehensibility problems. A general conclusion of the project 
was that high-quality training on specific procedures is generally 
superior to training limited to system knowledge. The developed 
feedback system was sophisticated and fast. Future work on this topic 
might best be focused cn documenting the .redundancy and 
insignificance of much mental model content. The reports and 
publications of this portion of the project are also listed. 
Applications of the work of both parts of the project and the 
problems encountered are summarized. (SLD) 
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ABSTRACT 

This is the final report for a two part project. Theoretical and empirical work was conducted on the topic 
of the role of "mental model," or how-it-works information, in teaming to operate equipment. A general 
conclusion is that high-quality training on specific procedures is generally superior to training limited to 
system knowledge, whk:h was relatively difficult to learn and to apply. Additional work was also done on 
learning procedures from text, wiiich further extended production-system models as an account of 
procedural teaming. The second part of the project was further work on a computer-based aid for 
comprehensible writing of technical materials. Appllcattons of the project worx and various problenss 
encountered are summarized. 

Project Goals 

This research contract actually encompassed two projects. The original oi.e, under the titte Learning 
about Equipment from Technical Documentation, was concerned with empirical and cognitive modeling 
studies of how people team to operate equipment from the kind of information contained in technical 
documentation, namely, information about operating procedures, and information about how the device 
works. The goal was to understand how knowledge about equipment couUS be effectively presented, 
both in terms of what the user Is supposed to do procedurally, and what about the equipment itself should 
be conveyed. The second portion of the project was added later for administrative reasons: this was a 
separate line of work on a computerized aid for comprohensibie writing, a computer program intended to 
provide feedback to the writers of technical documents concerning comprehensibility problems. This 
sub-project was sponsored by the Navy Personnel Research and Development Center. 



Work Accomplished 
The Role of Mental Model Infonnatlon 

Questions from earlier work. This project started where earlier wori< left off (scs Kieras & Bovair, 1 984) 
on the topic of the role of mental model information about how the device wort<s. At that point we had 
demonstrated empirically that training in mental model infonnation for a simple control panel device 
resulted in nxjch improved performance in paradigms where people were explicitly taught or had to infer 
procedures for operating the device. A simulation model had been constructed of the mental model 
reasoning process, (see Kieras, 1984; Kteras, in press) and some wort< was done during the period of this 
project showinc ♦hat the model accounted for s jme aspects of the data. This wori< was submitted to bvo 
different journal? with negative outcomes. Overall the reviewers dkl not seem to agree that the approach 
was worthwhile and did not consider that the comparison of the model to the data was adequately 
impressive. The data in questton were indivkiual inter-response latencies for elementary actions on the 
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device (e.g. pushing a button) where each subject where each subject contributed one trial on each 
problem. Since the model was a model only of correct performance, only about half of the response 
sequences in the data could be comparnd to the model. Qt these, quantitative predictions made from the 
model and various nuisance variables could account for roughly 40% of the variance in the inter-response 
latencies for individual subjects, which as it happens is close to the maximum variance that could be 
accounted for given the reliability of the data. Thus, it would appear that perhaps the reviewers simply did 
not appreciate that accounting for problem-solving data in such detail was an accomplishment, or perhaps 
there has been a shift in paradigm where simply being able to model some data is not considered 
publishable anymoro. But in any event, because new data were being collected, I feit that further effort to 
publish an admittedly limited set of results was not worthwhile compared to work using more 
comprehensive and reliable data sets. 

So a new line of studies was begun, with the idea that the modeling of mental model processes would 
be done in service of a larger scale goal, rather than just demonstrating that such models were possible. A 
set of such larger goals appeared during work on my contribution to the ONR- and ETS-sponsored 
conference on "Diagnostic monitoring of skill and knowledge acquisition' (Kieras, in press). The mental 
models trained in the eariler studies were In fact logicaily incomplete, and contained highly specific rather 
than gefieric information. This differed sharply from the assumptions that 1 had made in the cognitive 
simulation model, and so it was clear that the content of the mental model training did not agree with the 
logtoaily complete specificatton of the mental model represented by the simulation. Periiaps a cleaner set 
of data, and better fit of the model wouki be obtained under conditions where subjects were trained in the 
same mental model content as the simulation. 

Furthermore, during the same wort< I realized that the eariler studies had compared mental model 
training to conditions in which subjects either intended the procedures on a trial and envr basis, or were 
given training in rather sub-optimal procedures. That is, the or^iinai study in Kieras and Bovair (1984, Exp. 
1) trained subjects in procedures that were highly overiapping and highly redundant, but this was not 
made explicit to the subjects. Our transfer wori< (Kieras and Bovair, 1986) was based on the realization 
that this overlap explained the rote training data. Thus, the procedure training that subjects received was 
rather sub-optimal. It tums out that a complete and accurate procedure for the control panel device can be 
stated in almost the same number of steps as just one of the ten procedures that subjects were trained 
on. 

So. a better view of the Kieras and Bovair (1984) studies is that mental model training was shown to be 
superior to both trial and error and grossly inforier procedure training. The problem is that military training 
practkjes correspond to neither one of these radically inferior conditions, however much their need for 
improvement. The new studies compared mental nrx)del training to hlgh-qualltypToce6ure training. 

A final goal of the new line of studies was to get performance up to a higher level than previously so 
that a larger subset of the behavior sequences could be compared to the simulation models. 

New control panel device studies. A series of three careful studies on the control panel device were 
undertal<en; these studies were quite sound methodologlcaily and produced very high-quality data. In 
these studies different fornis of mental model training were compared to a high-quality rote procedure 
training condition. The results were ambiguous for understanding the value of mental nxjdei-based 
training. Throughout all of the studies, the rote procedure training was by far the best condition in terms of 
speed of learning and quality of perfomiance. The different mental model conditions proved to be 
indistinguishable, except in the eariiest trials, generally poor in overall learning, and showed a rapid 
convergence to the same execution time characteristics as the rote procedure condition. 



One of the goals for the studies was to demonstrate thai more complete and comprehensive mental 
model training would be superior to the sl<etchler training used in the original studies. Thus, the best 
condition was predicted to be one in which subjects studied the same generic inference rules and 
strategy as in the original simulation model. But this tumed out to be the worst condition! Overall, the 
results were that the more complete and elaborate the mental model training, the longer It tool< people to 
learn it and to learn how to apply It. 

A second goal was to get more stable, reliable time data by having the subjects repeat the problems, 
rather than solve them just one time, as before. Most frustrating was that initially, the training conditions 
could be distinguished, but performance was poor, and as the subjects gained experience, performance 
improved substantially, but the training conditions became indistinguishable. Thus the only effects of the 
mental model training wer^^ in the first few trials, and on these trials, the perfomnance was generally no 
better in these experiments than they had been with the original ones. The procedure training condition 
was superior on these initial trials, and remained at least equal to all other conditions thereafter. 

In summary, these studies suggest that subjects in the mental nrK>del conditions quickly construct a 
procedureior operating the device; they do this by making inferences from the mental model training 
materials, which can be dKftoult to do depending on the amount, complexity, and abstractness of the 
material. On the other hand, subjects given the procedure directly are far better off. Hence, acquiring 
procedures from text is superior to infening them from a mental model. 

Scaled'Up mdntal model study. Another study was done in whk:h the effects originally observed with 
the control pane. .•'9vk5e were sought in a scaled-up version. A computer-based version of the control 
panel devtoe system was defined and implemented. Instead of operating switches and observing 
indtoator lights, the subjects typed in commands and viewed status information on a vkJeo terminal. The 
three training conditions compared were (1) rote procedure training; (2) addittonal training in the syntax 
and semanttes of the command language, which is nfwst like tradittonal computer training; and (3) 
additional training on the underlying dynamic characteristics of the system, which directly supported 
inferences of correct operating procedures for novel situations. 

The resuSts showed a moderately strong overall benefit of more complete training, but this effect, 
contrary to ex,3ectation, failed to be speclfte to the indivkjual problem solving situations. Furthermore, in a 
post /70C analysis of the procedural content of the training and testing, it appeared that to a good first 
approximation, the time taken to execute the task was a function of how much procedural knowledge the 
subjects had acquired in their previous experience with the devtoe. This very powerful effect apparently 
masked many of the benefits that training in system knowledge might have provWed. 

Conclusions on mental model training. The overall concluston from these studies is that high-quality 
training on procedures is probably the most effteient training approach. We have yet to see a ca. j where 
training in mental model content is of definite benefit above and beyond high-quality procedure training. 
Perhaps there are situations where good procedure training is impractical. But otherwise, these studies 
imply that the benefits of understanding how a system wori<s will be limited to very transitory first-time or 
one-of-a-kind situattons. Once a subject has successfully problem-solved through a situation, the mental 
model content is of little value thereafter. These results are being prepared as a techntoal report. 

Certainly further wortt on the value of mental models in training is needed, but we shoukl probably be 
prepared for the outcome of this research being very different from what we woukJ have prevtously 
imagined. For exanrpie. It could be that most mental model training in military equipment systems Is simply 
a waste of time, given that in the actual wori< situation the procedures for interacting with nwst equipment 
have been made eyplicit, and when these procedures are not adequate to cover the situatton, there is 
little that can be done other than to wait for an expert to repair the equipment. An example of this appears 



in some of the actual Navy training and job aid materiais that i have examined, in which even emergency 
malfunction procedures are speiied out in considerable detail. In contrast, the training includes classroom 
and textbook principles of systems such as steam turbines that are dwelt on at some length, even though 
nowhere in the trainee's job (or several grades above it) could this Icnowledge ever possibly be applied. 
Perhaps the major effect of such training is only to convey terminology and a knowledge of system 
components (e.g. types of valves and" bearings) and certain elementary basic principles, such as why 
bearings can not be allowed to get too hot. Thus future wori< on this topic might best be focussed on 
documenting the redundancy and insignifteance of much mental model content for many military jobs, 
rather than further attempts to demonstrate its usefulness. 



Acquiring Proceduns from Toxt 



This wort< was carried on mostly by Susan Bovair. Extensions of this wort; and the simulation modeling 
of the data will be the subject of her dissertation. 

Major papers. Two major papers on this topic were published with Bovair first author on both; the first 
was accepted at Human-Computer Interaction, with the data originally collected under IBM sponsorship. 
ONR support for the preparation of this r aper was acknowledged. This paper concerned the learning and 
execution of procedures for operating a text editor. The second paper was an important theoretical paper 
on the acquisition of procedures from text for the new Handbook of Reading Research. This is a key 
paper because it is the first time a theorettoally sophisticated treatment of this topks has been prepared and 
presented in a place where a variety of reading researchers will see it. One functton of the paper is to 
emphasize the lack of research in this area: It is drastteally under-exptored. The paper also presents our 
framewort< for a model of how procedures are acquired, based on the construction of production rule 
representattons for procedures using the infomiation presented In the text. The preparation of this paper 
was an Important activity In this project. 

'Vverload" study. Susan Bovair conducted an experiment on the "overtoad effect" noted in our 
eariler wort< on procedure acquisition. In this eariier experiment (Kieras and Bovair. 1986) we observed 
that one of the procedures, when it was the first one learned, took much more time than was predicted by 
the production rule nwdei. Our hypothesis was that this was due to a wortdng memory overtoad during 
procedure leaming; this one procedure when learned first involved a total number of productkjn rules that 
was substantially larger than that in any other situatton in the experiment. If a representation of the entire 
procedure had to be constaicted in wori<lng memory, then subjects presumably would have to engage in 
consWerable extra processing to rnake up for the fact that they coukj not maintain the entire procedure at 
once in memory. This hypothesis vas tested with a very carefully designed experiment that manipulated 
the degree of overioad with a complex set of procedures defined on the control panel device. The 
experiment was designed using the simulation model of procedure acquisltton to predict the number of 
new rules and overioad effect a priori. A set of procedures and a set of training orders was defined so as to 
produce a predicted overtoad effect under conditions where it wouW not be confounded with other 
factors, such as being the very first trial in the experiment. 

The basic results were that the putative overtoad effect occurred, but only eariy In the experiment- no 
overioad effect occurred in the second half of the procedures. This initially very puzzling result fell into 
place when, as a result of the thinking involved in preparing the review paper. Bovair realized that there are 
a whole set of low-level procedures that subjects woukJ have to leam in the experiment. For example, if 
one of the steps In the procedure was Set S2 to X then the subjects had to leam a whole set of 
productton rules for carrying out what had appeared to us to be a single step. For example, first they have 
to kx:ate the control S2. then grasp It. then rotate It in one directton or the other, and stop when the 
pointer is at the positton X. Thus, there Is In fact a set of lower-level productton mies that must be teamed 



in order to operate the equipment. Since these sane basic activities are used repeatedly in all of the 
procedures, they only have to be learned early in the experiment. 

Bovair was able to describe these low-level procedures and make quantitative predictions of training 
time using this expanded set of production ruless. The fit to the data is very good. She has reanalyzed our 
earlier data from Kieras and Bovair (1986) with similar results. These results are also being prepared as a 
technical report. 

Conclusions on procedure acquisition. The work done In this project on procedure acquisition has 
been largely theoretical, but with some Important empirical results that show that the basic approach of 
modeling procedure leaming In temts of the acquisition of production rules is both more robust and 
simpler than originally expected. Bovair's dissertation work, In which she will construct a comprehensive 
simulation model of the procedure acquisition process, shoukl be a key piece of v;ork in capturing the 
theoretical Insights that we have accumulated thus far. 

Comprehensiblllty System 

Several versions of the comprehensiblllty system were delivered to NPROC. This system has evolved 
considerably during Its tenure as part of this project. The parser is now very sophisticated and very fast, 
and the set of criticism rules incorporated In the model have expanded into a fairly large and 
comprehensive set. This software was also repeatedly revised and Improved both for efficiency and 
speed, but also for ease of future revision and updating. Additionally, the software was designed so that 
portions of it could be easily used In cognitive modeling projects such as Bovair's dissertation work. A 
chapter on the system appeared In the key Britton and Glynn volume on computer writing environments, 
ensuring Its visibility to a wider audience. 

At this point the major remaining problem with the comprehensiblllty system is that its output Is still too 
verbose. The last work to be done on this version is to complete final work in extensions to the grammar, 
and simplify the output. One more verston of the system is slated to be delivered to NPROC, along with a 
technical report that provides maintenance and extension Informatton for NPRDC staff. 



Reports and Publications 

Technical Reports 

Kleras, D.E. (1987). The role of cognitive simulation models in the devetopment of advanced training and 
testing systems (Tech. Rep. No. 23, TR-87/ONR-23). Ann Arbor: University of Michigan, Technical 
Communlcatton Program. (DTIC AD A1 78268) 

Kleras, D.E. (1987). What mental model shouW be taught: Choosing instmcliona: content for complex 
engineered systems (Tech. Rep. No. 24, TR-87/ONR-24). Ann Arbor: Unu i.sity of fi/lichigan. Technical 
Communtoation Program. (DTIC AD A1 783S2) 

Bovair, S., Kieras, D.E., & Poison, P.G. (1988). The acquisition and performance of text editing skill: A 
proGuction system analysis (Tech. Rep. No. 28). Ann Arbor: University of Michigan, Technical 
Communication Program. 

Bovair, S. i Kleras, D.E., (1989). Toward a model of acquiring procedures from text (Tech. Rep. No. 30, 
TR-89/ONR-30). Ann Arbor: University of Michigan, Technical Communlcatton Program. 
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Reports In Preparation 

Bovair, S. High- and low-level procedure transfer effects in learning to operate a device (Tech. Rep. No. 
32, TR-90-ONR-32). Ann Artwr: University of Michigan, Technical Communication Program. 

Kieras. D.E. Rote procedure versus mental model training in operating equipment (Tech Rep. No. 33, TR- 
90-ONR-33). Ann Arbor: University of Michigan, Technical Communication Program. 

Kieras, D.E. Programmers' guide for the computerized comprehensibiiity system (Tech. Rep. No. 34, TR- 
90-ONR-34). Ann Arbor: University of Michigan, Technical Communication Program. 



Publications 

Kieras, D.E. (1987). Cognitive Modelling. In Encyclopedia of Artificial Intelligence, New York: Wiley. 

Kieras, D.E. Mental models for engi leered systems and computers. In the proceedings of Workshop on 
the Role of Mental Models in User-Centered System Design, under Army Research Institute 
sponsorship, hosted by University of Colorado Institute for Cognitive Science, Breclcenrldge, CO, 
January 13-16, 1988. 

Kieras. D.E. (1988). What mental nrjodel should be taught: Choosing instructional content for complex 
engineered systems. In J. Psotka, L.D. Massey, and S. Mutter (Eds.), Inteil^ent Tutoring Systems: 
Lessons Learned (pp. 85-1 11). Hillsdaie. NJ: Eribaum. 

Bovair, S. & Kieras, D.E. (1989). Toward a model of acquiring procedures from text. In Barr, Kamu, 
Rosenthal and Pearson (Eds.), Handbool< of Reading Research, Vol. II. Longman 

Kieras, D.E. (1989). An advanced computerized aid for the writing of comprehensible technical 
documents. In B. Brrtton & S. Glynn (Eds.), Computer Writing Environments: Theory, Research, and 
Design. Hillsdale, NJ: Eribaum. 

Bovair, S., Kieras, D.E., & Poison, P.G. (In press). The acqulsitton and perfomiance of text editing skill: A 
cognitive complexity analysis. Human-Computer Interaction. 



Problems Encountered 

The productivity of this project was Impaired by many administrative and managerial problems. They 
are summarized here by way of explanation, and in hopes that they might be instmctive for future projects. 
This project was Initiated while I was still at the University of Arizona; then the project was moved to the 
Unlverslty of Mtohlgan. Of course there was the normal disruption due to the various problems of moving. 
But an especially serious one was that a new laboratory computer, a VAX 730, was Installed. Although 
supplied at no cost to the project, conskierable project resources during the first year were consumed in 
getting software converted and Implemented. Given the obsolescence of the machine at this time, this 
large effort Is not likely to pay off In the long mn. It appears at this time an almost Ideal environment for 
training experiments is a large screen Macintosh II with software packages such as HyperCard or Course of 
Action, whteh reduce the software programming requirements to a very low level, even though far more 
features and power are available. 
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Another computer-related problem was that at the time of the move, we had just received a set of 
Xerox 1 108 Al Workstations, and a substantial effort was made to Implement all cognitive modeling 
software on these machines and pursue other related work such as the comprehenslbllity system In this 
environment as well. These were very difficult machines to use, and consumed a very large amount of 
time, i-iowever, cur actual computational needs were much simpler; we simply needed a high 
performance LISP implementation, and many of the features of the workstations that made them useful for 
intelligent tutoriiig system work were simply a distraction for us. A very large amount of software work was 
discarded after these machines were replaced with general purpose workstations (Apollos) which have 
proved lo be far more satisfactory. 

The most serious problem was a massive over-commitment of the Principal Investigator. In addition to 
the complications of moving to a different university and dealing with new lab equipment, I had too many 
projects active during the period of this project. In additton to this project, I was P.I. on a major IBM- 
sponsored project, a medium-sized project sponsored by NASA, the original comprehensibility system 
project, an addittonal ONR project that started later, and a substantial applied project for NPRDC. This 
over-commitment resulted in a lack of focus and erratic management of the staff. The most serious 
manifestation was that I had to take over the programming for the comprehensibility system. A more 
substantive effect of this over-commitment is that there were many promising research leads that opeiied 
up during this piojoct that simply could not be followed up. 



Accomplishments 

Two items to mention under this heading involved aspects of this work makiny a transition towards 
applteatton. The first was that the research on mental nrxxleis was directly applied to work done for NASA 
on diagrammatto displays for engineered systems. The basic thmst was that diagrams, and especially 
computer-generated animated and color-coded diagrams, can be used to convey mental model 
information to the user. If this is done in a form that directly facilitates the inferences that the user needs to 
perfomi, problem solving ability with a system shouW be improved. These effects, atong with some 
important limitations, were demonstrated both for the simple control panel devtee, and also for a complex 
piece ictual spacecran equipment. The following is the reference for the report on this topic: 

Kieras, D.E., (1988). Diagrammatic displays for engineered systems: Effects on human performance in 
interacting with malfunctioning systems (Tech. Rep. No. 29). Ann Arbor: University of Michigan, 
Technteal Communication Program. 

At the request of Dr. QeraM Laabs of NPRDC, we conducted a sot of analyses of a test battery for job 
performance. These tests consisted of acnuai procedure executions for a variety of tasks involving ship 
machinery. Our analysis deait with both the procedural and mental model content of the tasks and their 
relationship to the background knowledge and training of the personnel. The result was a suggestion for 
simple methods for ensuring non-redundant task selection. The reference for the report on this project is 
as follows: 

Kieras, D.E. A cognitive analysis of the relations between a set of hands-on job performance test tasks. 
Tech. Rep. prepared for the Navy Personnel Research and Development Center, Gerald Laabs, 
Scientific Monitor, under Sclentifk; Sen/ices Program Contract No. DAAL03-86-D-0001, Delivery Order 
694. March 3, 1989. 
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